Search for: All records

World model based reinforcement learning (RL) has emerged as a promising approach for autonomous driving, which learns a latent dynamics model and uses it to train a planning policy. To speed up the learning process, the pretrain-finetune paradigm is often used, where online RL is initialized by a pretrained model and a policy learned offline. However, naively performing such initialization in RL may result in dramatic performance degradation during the online interactions in the new task. To tackle this challenge, we first analyze the performance degradation and identify two primary root causes therein: the mismatch of the planning policy and the mismatch of the dynamics model, due to distribution shift. We further analyze the effects of these factors on performance degradation during finetuning, and our findings reveal that the choice of finetuning strategies plays a pivotal role in mitigating these effects. We then introduce AdaWM, an Adaptive World Model based planning method, featuring two key steps: (a) mismatch identification, which quantifies the mismatches and informs the finetuning strategy, and (b) alignment-driven finetuning, which selectively updates either the policy or the model as needed using efficient low-rank updates. Extensive experiments on the challenging CARLA driving tasks demonstrate that AdaWM significantly improves the finetuning process, resulting in more robust and efficient . 

World model based reinforcement learning (RL) has emerged as a promising approach for autonomous driving, which learns a latent dynamics model and uses it to train a planning policy. To speed up the learning process, the pretrain-finetune paradigm is often used, where online RL is initialized by a pretrained model and a policy learned offline. However, naively performing such initialization in RL may result in dramatic performance degradation during the online interactions in the new task. To tackle this challenge, we first analyze the performance degradation and identify two primary root causes therein: the mismatch of the planning policy and the mismatch of the dynamics model, due to distribution shift. We further analyze the effects of these factors on performance degradation during finetuning, and our findings reveal that the choice of finetuning strategies plays a pivotal role in mitigating these effects. We then introduce AdaWM, an Adaptive World Model based planning method, featuring two key steps: (a) mismatch identification, which quantifies the mismatches and informs the finetuning strategy, and (b) alignment-driven finetuning, which selectively updates either the policy or the model as needed using efficient low-rank updates. Extensive experiments on the challenging CARLA driving tasks demonstrate that AdaWM significantly improves the finetuning process, resulting in more robust and efficient . 

Deep reinforcement learning (deep RL) has emerged as an effective tool for developing controllers for legged robots. However, vanilla deep RL often requires a tremendous amount of training samples and is not feasible for achieving robust behaviors. Instead, researchers have investigated a novel policy architecture by incorporating human experts' knowledge, such as Policies Modulating Trajectory Generators (PMTG). This architecture builds a recurrent control loop by combining a parametric trajectory generator (TG) and a feedback policy network to achieve more robust behaviors. In this work, we propose Policies Modulating Finite State Machine (PM-FSM) by replacing TGs with contact-aware finite state machines (FSM), which offers more flexible control of each leg. This invention offers an explicit notion of contact events to the policy to negotiate unexpected perturbations. We demonstrated that the proposed architecture could achieve more robust behaviors in various scenarios, such as challenging terrains or external perturbations, on both simulated and real robots. 

Electronic devicesforrecording neuralactivityinthe nervoussyste m needto bescalableacrosslargespatialandte mporalscales whilealso providing millisecondandsingle-cellspatiote mporalresolution. H o w e v e r, e xi s ti n g hi g h- r e s ol u ti o n n e u r al r e c o r di n g d e vi c e s c a n n o t achievesi multaneousscalability on bothspatialandte mporallevels due toatrade-offbetweensensordensityand mechanicalflexibility. Here weintroduceathree-di mensional(3D)stackingi mplantableelectronic platfor m,basedonperfluorinateddielectricelasto mersandtissue-levelsoft multilayerelectrodes,thatenablesspatiote mporallyscalablesingle-cell neuralelectrophysiologyinthenervoussyste m. Ourelasto mersexhibit stable dielectric perfor mancefor overayearin physiologicalsolutions andare10,000ti messofterthanconventional plastic dielectrics. By leveragingthese uniquecharacteristics we developthe packaging of lithographednano metre-thickelectrodearraysina3Dconfiguration with across-sectionaldensityof7.6electrodesper100μ m2.Theresulting3D integrated multilayersoftelectrodearrayretainstissue-levelflexibility, reducingchronici m muneresponsesin mouse neuraltissues,and de monstratestheabilitytoreliablytrackelectricalactivityinthe mouse brain orspinalcord over months without disruptingani mal behaviour. 

Clinical translation of stem cell therapies for heart disease requires electrical integration of transplanted cardiomyocytes. Generation of electrically matured human induced pluripotent stem cell–derived cardiomyocytes (hiPSC-CMs) is critical for electrical integration. Here, we found that hiPSC-derived endothelial cells (hiPSC-ECs) promoted the expression of selected maturation markers in hiPSC-CMs. Using tissue-embedded stretchable mesh nanoelectronics, we achieved a long-term stable map of human three-dimensional (3D) cardiac microtissue electrical activity. The results revealed that hiPSC-ECs accelerated the electrical maturation of hiPSC-CMs in 3D cardiac microtissues. Machine learning–based pseudotime trajectory inference of cardiomyocyte electrical signals further revealed the electrical phenotypic transition path during development. Guided by the electrical recording data, single-cell RNA sequencing identified that hiPSC-ECs promoted cardiomyocyte subpopulations with a more mature phenotype, and multiple ligand-receptor interactions were up-regulated between hiPSC-ECs and hiPSC-CMs, revealing a coordinated multifactorial mechanism of hiPSC-CM electrical maturation. Collectively, these findings show that hiPSC-ECs drive hiPSC-CM electrical maturation via multiple intercellular pathways. 

Abstract Human induced pluripotent stem cell derived brain organoids have shown great potential for studies of human brain development and neurological disorders. However, quantifying the evolution of the electrical properties of brain organoids during development is currently limited by the measurement techniques, which cannot provide long‐term stable 3D bioelectrical interfaces with developing brain organoids. Here, a cyborg brain organoid platform is reported, in which “tissue‐like” stretchable mesh nanoelectronics are designed to match the mechanical properties of brain organoids and to be folded by the organogenetic process of progenitor or stem cells, distributing stretchable electrode arrays across the 3D organoids. The tissue‐wide integrated stretchable electrode arrays show no interruption to brain organoid development, adapt to the volume and morphological changes during brain organoid organogenesis, and provide long‐term stable electrical contacts with neurons within brain organoids during development. The seamless and noninvasive coupling of electrodes to neurons enables long‐term stable, continuous recording and captures the emergence of single‐cell action potentials from early‐stage brain organoid development.